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INTRODUCTION 


Two  general  types  of  inhibition  mediate  activity  in  neural  systems:  subtractive 
inhibition,  which  sets  a  zero  level  for  the  computation,  and  multiplicative  (nonlin¬ 
ear)  inhibition,  which  regulates  the  gain  of  the  computation.  We  report  a  physical 
realization  of  general  nonlinear  inhibition  in  its  extreme  form,  known  as  winner- 
take-all. 

We  have  designed  and  fabricated  a  series  of  compact,  completely  functional 
CMOS  integrated  circuits  that  realize  the  winner-take-all  function,  using  the  full 
analog  nature  of  the  medium.  This  circuit  has  been  used  successfully  as  a  component 
in  several  VLSI  sensory  systems,  that  perform  auditory  localization  (Lazzaro  and 
Mead,  in  press)  and  visual  stereopsis  (Mahowald  and  Delbruck,  1988).  Winner- 
take-all  circuits  with  over  170  inputs  function  correctly  in  these  sensory  systems. 

We  have  also  modified  this  global  winner-take-all  circuit,  realizing  a  circuit 
that  computes  local  nonlinear  inhibition.  The  circuit  allows  multiple  winners  in 
the  network,  and  is  well  suited  for  use  in  systems  that  represent  a  feature  space 
topographically  and  that  process  several  features  in  parallel.  We  have  designed, 
fabricated,  and  tested  a  CMOS  integrated  circuit  that  computes  locally  the  winner- 
take-all  function  of  spatially  ordered  input. 


THE  WINNER- TAKE- ALL  CIRCUIT 


Figure  1  is  a  schematic  diagram  of  the  winner-take-all  circuit.  A  single  wire, 
associated  with  the  potential  Vc,  computes  the  inhibition  for  the  entire  circuit;  for 
an  n  neuron  circuit,  this  wire  is  0(n)  long.  To  compute  the  global  inhibition, 
each  neuron  k  contributes  a  current  onto  this  common  wire,  using  transistor  T^k. 
To  apply  this  global  inhibition  locally,  each  neuron  responds  to  the  common  wire 
voltage  Vc,  using  transistor  T\k.  This  computation  is  continuous  in  time;  no  clocks 
are  used.  The  circuit  exhibits  no  hysteresis,  and  operates  with  a  time  constant 
related  to  the  size  of  the  largest  input.  The  output  representation  of  the  circuit  is 
not  binary;  the  winning  output  encodes  the  logarithm  of  its  associated  input. 
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Figure  1  Schematic  diagram  of  the  winner-take-all  circuit.  Each  neuron  receives  a  unidirec¬ 
tional  current  input  /&;  the  output  voltages  Vi . . .  Vn  represent  the  result  of  the  winner-take-all 
computation.  If  ifc  =  max^  .. .  In),  then  Vfc  is  a  logarithmic  function  of  I}.\  if  /,•  <C  1^,  then 

Vj  «  0. 

A  static  and  dynamic  analysis  of  the  two-neuron  circuit  illustrates  these  system 
properties.  Figure  2  shows  a  schematic  diagram  of  a  two-neuron  winner-take-all 
circuit.  To  understand  the  behavior  of  the  circuit,  we  first  consider  the  input 
condition  I\  =  I2  =  Im.  Transistors  Ti±  and  T\2  have  identical  potentials  at  gate 
and  source,  and  are  both  sinking  Im\  thus,  the  drain  potentials  V\  and  V2  must  be 
equal.  Transistors  r2l  and  T%2  have  identical  source,  drain,  and  gate  potentials, 
and  therefore  must  sink  the  identical  current  ICl  =  IC2  =  Ic/2.  In  the  subthreshold 
region  of  operation,  the  equation  Im  =  I0exp(Vc/V0)  describes  transistors  Ti1  and 
Xi2,  where  I0  is  a  fabrication  parameter,  and  V0  =  kT/qK.  Likewise,  the  equation 
Ic/2  =  I0  exp((Vm  —  Vc)/V0),  where  Vm  =  V\  —  V2,  describes  transistors  T2l  and 
T2<2 .  Solving  for  Vm(Im,Ic)  yields 


Vm  =  V„ln(^)+V„ln(A).  (1) 

±0  £lo 

Thus,  for  equal  input  currents,  the  circuit  produces  equal  output  voltages;  this 
behavior  is  desirable  for  a  winner-take-all  circuit.  In  addition,  the  output  voltage 
Vm  logarithmically  encodes  the  magnitude  of  the  input  current  Im. 
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Figure  2  Schematic  diagram  of  a  two-neuron  winner-take-all  circuit. 

The  input  condition  I\  =  Im  +  6i,  I2  =  Im  illustrates  the  inhibitory  action  of  the 
circuit.  Transistor  T\x  must  sink  more  current  than  in  the  previous  example;  as  a 
result,  the  gate  voltage  of  rises.  Transistors  T1:  and  Ti2  share  a  common  gate, 
however;  thus,  T\2  must  also  sink  Im  +  <5,-.  But  only  Im  is  present  at  the  drain  of 
T\2 .  To  compensate,  the  drain  voltage  of  Tl2,  V2,  must  decrease.  For  small  6j ;s,  the 
Early  effect  serves  to  decrease  the  current  through  T\2 ,  decreasing  V2  linearly  with 
Si.  For  large  S% ;s,  T\2  must  leave  saturation,  driving  V2  to  approximately  0  volts. 
As  desired,  the  output  associated  with  the  smaller  input  diminishes.  For  large 
let  «  0,  and  ICl  «  Ie.  The  equation  Im  +  =  I0exp(Vc/V0 )  describes  transistor 

T\x ,  and  the  equation  Ic  =  I0  exp((Vi  —  Vc) /V0)  describes  transistor  T2l .  Solving  for 
V\  yields 

Vi  =  V0  ln(^-^)  +  Vg  ln(^).  (2) 

10  1 0 

The  winning  output  encodes  the  logarithm  of  the  associated  input.  The  symmetrical 
circuit  topology  ensures  similar  behavior  for  increases  in  I2  relative  to  I\. 

Equation  2  predicts  the  winning  response  of  the  circuit;  a  more  complex  ex¬ 
pression,  derived  in  Appendix  A,  predicts  the  losing  and  crossover  response  of  the 
circuit.  Figure  3  is  a  plot  of  this  analysis,  fit  to  experimental  data. 
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Figure  3  Experimental  data  (circles)  and  theoretical  statements  (solid  lines)  for  a  two-neuron 
winner-take-all  circuit.  Jj,  the  input  current  of  the  first  neuron,  is  swept  about  the  value  of  J2>  the 
input  current  of  the  second  neuron;  neuron  voltage  outputs  Vi  and  V2  are  plotted  versus  normalized 
input  current. 

Figure  4  shows  the  wide  dynamic  range  and  logarithmic  properties  of  the  circuit; 
the  experiment  in  Figure  3  is  repeated  for  several  values  of  I2,  ranging  over  four 
orders  of  magnitude. 


Figure  4  The  experiment  of  Figure  3  is  repeated  for  several  values  of  J2;  experimental  data  of 
output  voltage  response  are  plotted  versus  absolute  input  current  on  a  log  scale.  The  output  voltage 
Vi  =  V2  is  highlighted  with  a  circle  for  each  experiment.  The  dashed  line  is  a  theoretical  expression 
confirming  logarithmic  behavior  over  four  orders  of  magnitude  (Equation  1). 
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The  conductance  of  transistors  T\x  and  T\2  determines  the  losing  response  of 
the  circuit.  The  Early  voltage,  Ve,  is  a  measure  of  the  conductance  of  a  saturated 
MOS  transistor.  The  expression 


dVd 

dL 


(3) 


defines  the  Early  voltage,  where  V&  is  the  drain  potential  of  a  transistor,  and  L  is  the 
channel  length  of  a  transistor.  Thus,  the  width  of  the  losing  response  of  the  circuit 
depends  on  the  channel  length  of  transistors  Tit  and  T\2.  Figure  3  shows  data  for 
a  circuit  where  the  channel  length  of  transistors  T\x  and  T\2  is  13.5/im.  Figure  5 
shows  data  for  a  circuit  with  a  wider  losing  response;  in  this  circuit,  the  channel 
length  for  transistors  Ti1  and  Ti2  is  3/xm,  the  smallest  allowable  in  the  fabrication 
technology  used. 
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Figure  5  Experimental  data  (circles)  and  theoretical  statements  (solid  lines)  for  a  two-neuron 
winner-take-all  circuit  with  a  channel  length  for  transistors  Tlx  and  Tls  of  3  #im.  The  dotted  lines 
show  the  losing  response  for  the  circuit  used  in  Figure  3,  which  has  a  channel  length  for  transistors 
Ti,  and  Txt  of  13.5  pm. 


Increasing  the  channel  length  of  transistors  Ti1  and  T\2  narrows  the  losing 
response  of  the  circuit;  alternatively,  circuit  modification  also  can  narrow  the  losing 
response.  The  circuit  shown  in  Figure  6  approximately  halves  the  width  of  the 
original  losing  response,  through  source  degeneration  of  transistors  T\x  and  Ti2  by 
the  added  diode-connected  transistors  T$x  and  T32.  Figure  7  shows  experimental 
data  for  this  modified  circuit. 


5 


Figure  6  Schematic  diagram  of  a  two-neuron  winner-take-all  circuit,  modified  to  produce  a 
narrower  losing  response. 


h/h 


Figure  7  Experimental  data  (circles)  and  theoretical  statements  (solid  lines)  for  a  two-neuron 
winner-take-all  circuit,  modified  to  produce  a  narrower  losing  response.  The  dotted  lines  show  losing 
response  for  the  circuit  used  in  Figure  4. 
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TIME  RESPONSE  OF  THE  WINNER- TAKE- ALL  CIRCUIT 


A  good  winner-take-all  circuit  should  be  stable,  and  should  not  exhibit  damped 
oscillations  (“ringing”)  in  response  to  input  changes.  This  section  explores  these 
dynamic  properties  of  our  winner-take-all  circuit,  and  predicts  the  temporal  re¬ 
sponse  of  the  circuit.  Figure  8  shows  the  two-neuron  winner-take-all  circuit,  with 
capacitances  added  to  model  dynamic  behavior. 


Figure  8  Schematic  diagram  of  a  two-neuron  winner-take-all  circuit,  with  capacitances  added 
for  dynamic  analysis.  C  is  a  large  MOS  capacitor  added  to  each  neuron  for  smoothing;  Cc  models 
the  parasitic  capacitance  contributed  by  the  gates  of  Tu  and  2\2,  the  drains  of  T2 1  and  T22)  and 
the  interconnect. 

Appendix  B  shows  a  small-signal  analysis  of  this  circuit.  The  transfer  function 
for  the  circuit  has  real  poles,  and  thus  the  circuit  is  stable  and  does  not  ring,  if 
Ic  >  4 1(Cc/C),  where  Ii  «  1%  «  I.  Figure  9  compares  this  bound  with  experimental 
data. 

If  Ic  >  4 1(Cc/C),  the  circuit  exhibits  first-order  behavior.  The  time  constant 
CV0j I  sets  the  dynamics  of  the  winning  neuron,  where  V0  =  kT/qK  «  40  mV.  The 
time  constant  CVe/J  sets  the  dynamics  of  the  losing  neuron,  where  Ve  «  50  V. 
Figure  10  compares  these  predictions  with  experimental  data,  for  several  variants 
of  the  winner-take-all  circuit. 
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Figure  9  Experimental  data  (circles)  and  theoretical  statements  (solid  line)  for  a  two-neuron 
winner-take-all  circuit,  showing  the  smallest  for  a  given  7,  necessary  for  a  first-order  response  to 
a  small-signal  step  input. 


Figure  10  Experimental  data  (symbols)  and  theoretical  statements  (solid  line)  for  a  two-neuron 
winner-take-all  circuit,  showing  the  time  constant  of  the  first-order  response  to  a  small-signal  step 
input.  The  winning  response  (filled  circles)  and  losing  response  (triangles)  of  a  winner-take-all 
circuit  with  the  static  response  of  Figure  3  are  shown;  the  time  constants  differ  by  several  orders  of 
magnitude.  Losing  responses  for  winner-take-all  circuits  with  the  static  responses  shown  in  Figure  5 
(squares)  and  Figure  7  (open  circles)  are  also  shown,  demonstrating  the  effect  of  the  width  of  static 
response  on  dynamic  behavior. 
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THE  LOCAL  NONLINEAR  INHIBITION  CIRCUIT 


The  winner-take-all  circuit  in  Figure  1,  as  previously  explained,  locates  the 
largest  input  to  the  circuit.  Figure  11  shows  this  behavior.  Figure  11(a)  is  the 
spatial  input  to  a  winner-take-all  circuit  with  16  neurons,  with  input  8  much  higher 
than  all  other  inputs.  Figure  11(b)  shows  the  circuit  response  to  this  input;  only 
neuron  8  has  significant  response. 

Certain  applications  require  a  gentler  form  of  nonlinear  inhibition.  Sometimes, 
a  circuit  that  can  represent  multiple  intensity  scales  is  necessary.  Without  circuit 
modification,  the  winner-take-all  circuit  in  Figure  1  can  perform  this  task.  Appendix 
C  explains  this  mode  of  operation. 

Other  applications  require  a  local  winner-take-all  computation,  with  each  win¬ 
ner  having  influence  over  only  a  limited  spatial  area.  Figure  11(c)  shows  the  desired 
computation.  As  in  Figure  11(b),  neuron  8  has  the  largest  response  in  the  circuit. 
However,  neuron  8  suppresses  the  output  of  only  nearby  neurons;  neurons  far  from 
neuron  8  have  significant  responses,  encoding  their  input  signals. 


— i — i — i — i — i — i — i — i 
0  2  4  6  8  10  12  14  16 
k  (Position) 
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k  (Position) 
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Figure  11  Comparison  of  idealized  winner-take-all  spatial  response  and  the  desired  local 
winner-take-all  response.  The  horizontal  axis  of  each  plot  represents  spatial  position  in  a  16-neuron 
network,  (a)  The  plot  shows  a  spatial  impulse  function,  used  as  input  to  compare  the  two  concepts. 
The  vertical  axis  shows  the  input  current  to  each  neuron,  with  I8  »  Ik^ 8.  (b)  The  plot  shows  the 
winner-take-all  response,  (c)  The  plot  shows  the  local  winner-take- all  response,  show  neuron  voltage 
output  on  the  vertical  axis. 

Figure  12  shows  a  circuit  that  computes  the  local  winner-take-all  function.  The 
circuit  is  identical  to  the  original  winner-take-all  circuit,  except  that  each  neuron 
connects  to  its  nearest  neighbors  with  a  nonlinear  resistor  circuit  (Mead,  in  press). 
Each  resistor  conducts  a  current  Ir  in  response  to  a  voltage  AV  across  it,  where 

Ir  =  I,  tanh(AF/(2F0)).  (4) 

Ia,  the  saturating  current  of  the  resistor,  is  a  controllable  parameter.  The  current 
source  present  in  the  original  winner-take-all  circuit,  is  distributed  between  the 
resistors  in  the  local  winner-take-all  circuit. 
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Figure  12  Schematic  diagram  of  a  section  of  the  local  winner-take-all  circuit.  Each  neuron  t 
receives  a  unidirectional  current  input  /*;  the  output  voltages  V{  represent  the  result  of  the  local 
winner-take-all  computation. 

To  understand  the  operation  of  the  local  winner-take-all  circuit,  we  consider  the 
circuit  response  to  a  spatial  impulse,  defined  as  I*  »  I,  where  I  =  I^k.  Ik  >  Ik_x 
and  Ik  »  Ik+u  so  VCk  is  much  larger  than  Vck.  a  and  and  the  resistor  circuits 

connecting  neuron  k  with  neuron  k  —  1  and  neuron  A:  +  1  saturate.  Each  resistor 
sinks  Is  current  when  saturated;  transistor  T2fc  thus  conducts  21  s  +  Ic  current. 
In  the  subthreshold  region  of  operation,  the  equation  Ik  =  I0  exp(VCfc/V0)  describes 
transistor  Tik,  and  the  equation  21  s  +  IC  =  I0  exp((Fjt  —  VCk) /V0)  describes  transistor 
T2fc.  Solving  for  Vk  yields 

Vjt  =  V0  ln((2 h  +  Ic)/Io)  +  V0  Hh/Io).  (5) 

As  in  the  original  winner-take-all  circuit,  the  output  of  a  winning  neuron  encodes 
the  logarithm  of  that  neuron’s  associated  input. 

As  mentioned,  the  resistor  circuit  connecting  neuron  k  with  neuron  k— 1  sinks  Is 
current.  The  current  sources  Ic  associated  with  neurons  A:  — 1,  A:  —  2,  ...  must  supply 
this  current.  If  the  current  source  Ic  for  neuron  A;  —  1  supplies  part  of  this  current, 
the  transistor  T2fc_1  carries  no  current,  and  the  neuron  output  Vk_x  approaches  zero. 
Similar  reasoning  applies  to  neurons  fc  +  l,A;  +  2,  ....  In  this  way,  a  winning  neuron 
inhibits  its  neighboring  neurons. 

This  inhibitory  action  does  not  extend  throughout  the  network.  Neuron  k  needs 

only  Ia  current  from  neurons  k  —  1,  k  —  2, _  Thus,  neurons  sufficiently  distant 

from  neuron  k  maintain  the  service  of  their  current  source  Ic,  and  the  outputs  of 
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these  distant  neurons  can  be  active.  Since,  for  a  spatial  impulse,  all  neurons  k  —  1, 
k  —  2,  ...  have  an  equal  input  current  I,  all  distant  neurons  have  the  equal  output 

=  V0  In (Ic/I0)  +  V0  In (I/I0).  (6) 

Similar  reasoning  applies  for  neurons  k  +  1,  k  +  2,  ...  . 

The  relative  values  of  Is  and  Ic  determine  the  spatial  extent  of  the  inhibitory 
action.  Figure  13  shows  the  spatial  impulse  response  of  the  local  winner-take-all 
circuit,  for  different  settings  of  I3/Ic- 


k  (Position) 

Figure  13  Experimental  data  showing  the  spatial  impulse  response  of  the  local  winner- take- all 
circuit,  for  values  of  Ie/Ic  ranging  over  a  factor  of  12.7.  Wider  inhibitory  responses  correspond  to 
larger  ratios.  For  clarity,  the  plots  are  vertically  displaced  in  0.25  volt  increments. 


CONCLUSIONS 

The  circuits  described  in  this  paper  use  the  full  analog  nature  of  MOS  devices 
to  realize  an  interesting  class  of  neural  computations  efficiently.  The  circuits  exploit 
the  physics  of  the  medium  in  many  ways.  The  winner-take-all  circuit  uses  a  single 
wire  to  compute  and  communicate  inhibition  for  the  entire  circuit.  Transistor  T\k 
in  the  winner-take-all  circuit  uses  two  physical  phenomena  in  its  computation:  its 
exponential  current  function  encodes  the  logarithm  of  the  input,  and  the  finite 
conductance  of  the  transistor  defines  the  losing  output  response.  As  evolution 
exploits  all  the  physical  properties  of  neural  devices  to  optimize  system  performance, 
designers  of  synthetic  neural  systems  should  strive  to  harness  the  full  potential  of 
the  physics  of  their  media. 
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APPENDIX  A 

STATIC  RESPONSE  OF  THE  WINNER- TAKE- ALL  CIRCUIT 


Figure  3  in  the  main  text  compares  data  from  the  two-neuron  winner-take-all 
circuit  with  a  closed-form  theoretical  statement  describing  the  losing  and  crossover 
response  of  the  circuit.  This  appendix  derives  this  theoretical  statement. 

Figure  A1  shows  a  small-signal  circuit  model  of  the  two-neuron  winner-take-all 
circuit  (Figure  2  in  the  main  text).  For  a  particular  operating  point  [Ii,h,  ICl,  Ic2]> 
the  model  shows  the  effect  of  a  small  change  in  I\,  denoted  t’i,  on  the  circuit  voltages 
Vi,V2,  and  Vc,  indicated  by  the  small-signal  voltages  v\,v2,  and  vc.  In  this  model, 
a  linear  resistor  r^.,  in  parallel  with  a  linear  dependent  current  source,  with  a  con¬ 
ductance  ,  replaces  each  transistor  T{.  from  Figure  2.  For  a  particular  operating 
point  in  subthreshold,  the  small-signal  parameters  are 

011  =  h/V0  g2l  =  ICl/V0  rix  =  Ve/h  r2l  =  Ve/ICl 

9i2=h/V0  02  2  =  ICJV0  rl2  =  Ve/I2  r22=Ve/IC2,  [  ] 


where  Ve,  the  Early  voltage,  is  a  measure  of  transistor  resistance,  and  V0  =  JcT/qK. 
This  small-signal  model  is  a  linear  system,  which  we  can  solve  analytically  using 
conventional  techniques;  applying  the  approximation  Ve  +  V0  «  Ve  to  the  solution 
yields  the  simplified  equations 


S.  =  (i//1)(v0  +  ve(je2/ie)) 

*1 

J-  =  -v,(l  lh)(hjh). 

H 


9i2vc 


02iK  - 


922(V2-VC) 


Figure  A1  Small-signal  model  of  the  two-neuron  winner-take-all  circuit. 
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Note  that  both  small-  and  large-signal  quantities  appear  in  Equation  A2.  We 
can  view  the  small-signal  quantities  as  differential  elements  of  large-signal  quantities; 
as  a  result,  we  can  rewrite  Equation  A2  as  the  pair  of  nonlinear  differential  equations 


^  =  (1  lh)(Vo+V,(I'JIc)) 

^  =  -ve(i//i)(w/c). 


(.43) 


Solving  this  pair  of  nonlinear  differential  equations  yields  a  complete  description  of 
circuit  response.  We  begin  by  eliminating  Ic,  and  h2  from  the  equations.  Referring 
to  Figure  2,  the  equations 


Iei=J„ex p((Vi  -  Vc)/F„) 

/ej  =  /„exp((F2-Vc)/F„)  '  ’ 

describe  transistors  T2l  and  T%2.  From  Kirchoff’s  current  law,  we  know  that  Ic !  + 
Ic2  =  Ic  ;  substitution  of  Equation  A4  into  this  equation  yields  the  expression 

Ic  =  Io  exp  ((Vi  -  Vc)/V0)  + 10  exp((V2  -  Vc)/V0).  (A5) 


Dividing  Equation  A4  by  Equation  A5  eliminates  Vc,  leaving,  after  rearrangement, 


Ic\l  Ic  — 
IcJIc  = 


1 

l  +  exp((V2- VO/Vi) 

1 

1  +  exp  {{Vx-V^/VoY 


(A6) 


These  expressions  fit  nicely  into  Equation  A3,  eliminating  and  Ic2,  and  leaving 
a  set  of  differential  equations  involving  only  Vi ,  V2 ,  and  Ii : 


^=(i//.)(n+v.(1+exp((y;:y2)/n))) 

dh  =  +  exp((F2  -  V1)/Vi))' 

Equation  A7a  contains  V2  only  in  the  subexpression 

1 

l  +  exp((V1-V2)/V0)’ 

and  Equation  A7b  contains  V\  only  in  the  subexpression 


{Ala) 

(AU) 


(A8a) 


1 

1  +  exp((V2  —  Vi)/V0)  ’ 


(A86) 


These  subexpressions  are  both  Fermi  functions  of  the  difference  Vi  —  V2.  For  Vi  — 
V2  >  V0,  subexpression  A8a  is  approximately  zero,  whereas  subexpression  A8b  is 
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approximately  one;  for  V2  —  V\  ;§>  V0,  subexpression  A8a  is  approximately  one, 
whereas  subexpression  A8b  is  approximately  zero.  In  the  region  V\  «  F2,  we  can 
assume  V\  and  V2  are  both  changing  with  the  same  magnitude  of  slope  relative 
to  I\.  We  can  write  this  approximation  as  V\  —  V2  «  2  (Vi  —  Vm )  and  V2  —  V\  » 
2(V2  —  Vm),  where,  from  the  qualitative  analysis  in  the  main  text,  Vm  —  V\  =  V2 
when  Ii  =  I2  =  Im-  We  can  use  this  approximation  to  decouple  Equations  A7a  and 
A7b,  producing 


dV ! 
dlx 
dV2 

dli 


ta  )) 


~Ve(l/h){ 


l  +  exp(2(V2-Vm)/V0) 


)• 


(A9) 


We  can  solve  these  equations  by  straightforward  integration,  yielding,  after  appli¬ 
cation  of  the  approximation  Vg  +  V0  «  Ve, 

ln(VJm)  =  (Vi  -  Vm)/Ve  +  (1/2)  ln(l  +  (V0/Ve)  exp(2(F!  -  Vm) /Vo)){A10a) 
Hh/Im)  =  (Vm  -  V2)/Ve  +  (1/2)(V0/Ve)(l  -  exp(2(72  -  Vm)/V0)).  (AlOb) 


Equation  AlOa  predicts  the  value  of  Ji  for  a  given  value  of  Vi,  whereas  Equation 
A  10b  predicts  the  value  of  I\  for  a  given  value  of  V2;  in  this  way,  these  equations  are 
a  closed-form  approximation  of  circuit  response.  To  understand  the  behavior  of  the 
circuit,  and  to  evaluate  the  effect  of  the  approximations  V\  —  V2  «  2 (Vi  —  Vm)  and 
V2  —  Vi  «  2(F2  —  Vm),  we  can  simplify  Equations  A  10a  and  A  10b  for  three  regions 
of  interest:  V\  «  V2  »  Vm,  Vi  »  Vm  while  V2  <  Vm,  and  Vi  C  Vm  while  V2  »  Vm. 

First  consider  the  condition  V\  »  V2  «  Vm.  In  this  case,  |Vi  —  V2|  — »•  0,  I\jlm  -* 
1,  and  we  can  linearize  the  transcendental  functions  in  Equation’s  AlOa  and  AlOb, 
yielding  the  simpler  relations 


V1  =  (V./2)((71/im)-l))+Vm 

V>  =  (V,/2)(1  ~{h/Im))+Vm.  1  > 

In  this  region,  V\  and  V2  are  a  linear  function  of  I\,  with  a  slope  of  ±Ve/(2Im). 

Next,  consider  the  condition  Vx  »  Vm  while  V2  -C  Vm,  valid  when  Ii  >  Im. 
In  Equation  AlOb,  V2  «C  Vm  implies  exp(2(V2  —  Vm)/V0 )  -*■  0.  This  simplification 
yields,  after  rearrangement, 


V2  =  Vo/2  +  Vm  -  Vein (h/Im).  (A12) 

If  we  use  the  notation  ii  =  Im  +  Si,  as  in  the  earlier  qualitative  analysis,  we  can 
rewrite  the  subexpression  ln(Ii//m)  as  ln(l  +  (6,/Im)),  which  we  can  approximate 
as  6i/Im  for  small  (5,-//m,  yielding  the  simplified  result 


V2  =  Vo/2  +  Vm  -  (Ve/Im)Si. 


(-413) 
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Thus,  in  this  region,  V2  decreases  linearly  with  Si,  with  a  slope  of  Ve/Im,  which  is 
twice  as  large  as  in  the  previous  condition. 

We  can  similarly  derive  a  simplified  expression  for  Vi,  for  the  same  condition 
Vi  »  Vm  while  V2  <  Vm.  In  Equation  AlOa,  Vi  »  Vm  implies  (V0/Ve)exp(2(Vi  - 
Vm)/V0)  »  1.  This  approximation  yields,  after  rearrangement, 

Vi  =  V0  In (h/Im)  +  (Vo/2)  In {Ve/V0)  +  Vm.  (A14) 

For  this  condition,  as  predicted  earlier  in  Equation  2  in  the  main  text,  Vi  is  a  loga¬ 
rithmic  function  of  I\.  However,  when  does  the  approximation  (V0/Ve)  exp(2(Vi  — 
Vm) /Vo)  1  hold?  This  inequality,  when  rearranged,  yields  the  constraint 

(Vi  -  Vm)  »  (Vo/2)  In  (Vo/Vo).  (A15) 

Therefore,  for  a  typical  fabrication  process,  Vi—Vm  must  be  much  greater  than  0.15 
volts  for  Equation  A14  to  hold!  This  error  stems  from  the  central  approximation 
Vi  -  V2  «  2(Vi  -  Vm),  which  is  valid  for  only  T r\—V%  <  V0.  Thus,  for  this  region  of 
operation,  Equation  2  in  the  main  text  best  predicts  circuit  behavior. 

Finally,  we  consider  the  condition  Vi  <C  Vm  while  V2  >  Vm,  valid  when  Ji  < 
Im.  In  Equation  AlOa,  V\  <  Vm  implies  (V0/Ve)  exp(2(Fi  -  Vm)/V0)  — +  0.  This 
simplification  yields,  after  rearrangement, 

Vi=Vm  +  Veln  {h/Im).  (A16) 

If  we  use  the  notation  I\  =  Im  —  as  in  the  earlier  analysis,  we  can  rewrite  the 
subexpression  ln(Ii/Jm)  as  ln(l  —  (<S;/Jm)),  which  we  can  approximate  as  —6i/Im 
for  small  |  —  6t/Jm|,  yielding  the  simplified  result 

Vi=Vm-  ( Ve/Im)6i .  (A17) 

Thus,  in  this  region,  V\  decreases  linearly  with  with  a  slope  of  Ve/Im.  The  losing 
responses  for  V\  and  V2  are  thus  identical. 

We  can  similarly  derive  a  simplified  expression  for  Vjj,  for  the  same  condition 
V\  <C  Vm  while  V2  »  Vm.  For  Equation  A  10b,  V2  »  Vm  implies  exp(2(F2  — 
Vm)/V0)  »  1.  This  approximation  yields,  after  rearrangement, 

In {h/Im)  =  (Vm  -  V2)/Ve  -  (l/2)(V0/Ve)  exp(2 (V2  -  Vm)/V0).  (A18) 

As  V2  —  Vm  increases,  the  right  side  of  this  equation  grows  exponentially  large  and 
negative,  forcing  I\  to  grow  closer  and  closer  to  zero;  thus,  V2  is  constant  with  I\. 
However,  the  poor  approximation  V2  —  V\  «  2(V2  —  Vm )  for  V2—V\  >  V0  stunts  this 
exponential  growth.  The  qualitative  analysis  in  the  main  text  predicts  this  constant 
value  accurately,  as 

V2  =  V0  ln(^)  +  Vo  ln(f ).  (A19) 
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In  summary,  Equations  AlOa  and  A  10b  predict  the  losing  and  crossover  response 
of  the  circuit,  whereas  Equations  2  and  A 19  predict  the  winning  response  of  the 
circuit.  Figure  4  is  a  plot  of  this  analysis,  fitted  to  experimental  data.  Figure 
A2  expands  the  crossover  region  of  Figure  4,  showing  the  crossover  region  between 
losing  and  winning  analysis.  The  theoretical  predictions  in  Figure  5  and  Figure  7 
also  use  this  analysis,  with  altered  values  of  Ve. 


vuv2 

(V) 


h/h 


Figure  A2  Experimental  data  (circles)  and  theoretical  statements  (solid  lines)  for  a  two-neuron 
winner-take-all  circuit  in  the  crossover  region. 
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APPENDIX  B 

DYNAMIC  RESPONSE  OF  THE  WINNER-TAKE- ALL  CIRCUIT 


In  the  main  text,  we  presented  theoretical  predictions  of  the  time  response  of 
the  winner-take-all  circuit,  and  compared  these  predictions  with  experimental  data, 
in  Figure  9  and  10.  In  this  appendix,  we  derive  these  theoretical  predictions. 

Figure  8  in  the  main  text  shows  a  schematic  diagram  for  a  two-neuron  winner- 
take-all  circuit,  with  capacitances  added  to  model  dynamic  behavior.  Figure  Bl 
shows  a  small-signal  circuit  model  for  this  circuit. 


V\  V2 


Figure  Bl  Small-signal  model  of  the  two-neuron  winner-take-all  circuit,  with  capacitances  added 
to  model  dynamic  behavior. 


For  a  particular  operating  point  [Ii,l2,ICl,Ic2b  the  model  shows  the  effect  of  a 
small  change  in  Ii,  notated  t'i,  on  the  circuit  voltages  Vi, Vjj,  and  Vc,  indicated 
by  the  small-signal  voltages  Vi,V2,  and  vc.  In  this  model,  a  linear  resistor  rty,  in 
parallel  with  a  linear  dependent  current  source,  with  a  conductance  g^. ,  replaces 
each  transistor  T^.  from  Figure  2.  For  a  particular  operating  point  in  subthreshold, 
the  small-signal  parameters  are 

0U=h/V.  S2l=/C1/V,  ru  =  V./I,  r2l  =  V«//tl 

9i2  =  h/V„  g-l2  =  IJV0  r  la  =  V,/Jj  r2s  =  V,JIC2,  1  1 

where  Ve,  the  Early  voltage,  is  a  measure  of  transistor  resistance,  and  V0  =  kT /qK. 
This  small-signal  circuit  model  is  a  linear  system,  which  we  can  solve  analytically 
using  conventional  techniques.  The  resulting  solution,  unfortunately,  is  a  function 
of  the  unsolved  large  signal  ICl  and  IC2.  However,  for  the  input  conditions  I2  =  Im 
and  I\  =  Im  +  6i,  we  can  reasonably  make  the  approximations  Ic  and  0 
for  relatively  small  due  to  the  exponential  dependence  of  T21  and  T22  on  V\  and 
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Vi.  Using  these  approximations,  we  can  express  the  small-signal  voltages  v\  and  v% 
as  linear  functions  of  the  small-signal  input  current  i\ ,  as 


((CcV0/Jc)s  +  1) 


and 


where 


and 


V2 

H 


ui  _  (Y°\ 

h  ii  (s/(u  +  6)  +  l)(s/(a  -  6)  +  1) 

1 


4) 


h '  (( CVe/I2)s  +  l)(s/(a  +  b)  +  l){s/{a  -  b)  +  1) 


Ii  ^  Ic 

O'  — _ + 


2CVe  2  CcV0 


II  \o  ,  /  Ic  \o  t  Ich 


b  =  \/( — — ^ — )2  +  (^—  )2  -  ( 

\/  *  1  t/  /  v 


2(2^'  v2CcU0'  KCCcV£ 


)• 


(B2) 

(53) 

(54) 

(55) 


If  6  is  an  imaginary  number,  the  circuit  has  complex  poles,  and  exhibits  undesirable 
ringing  behavior.  If  Ic  >  4Ii(Cc/C),  then  b  is  real,  and  ringing  does  not  occur. 
Figure  9  in  the  main  text  compares  experimental  data  with  this  inequality. 


When  b  is  real,  the  circuit  exhibits  first-order  behavior.  We  can  simplify  Equa¬ 
tions  B2  and  B3,  and  show  that  the  first-order  time  constant  for  V\  is  CV0/I,  and 
the  first-order  time  constant  for  Vi  is  CVe  / 1,  where  Ii  «  Ii  =  I.  Figure  10  in  the 
main  text  compares  experimental  data  with  these  time  constants. 
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APPENDIX  C 

REPRESENTING  MULTIPLE  INTENSITY  SCALES 


This  appendix  explains  a  regime  of  operation  of  the  winner-take-all  circuit  that 
represents  multiple  input  intensity  scales  in  the  output,  while  still  functioning  as 
an  inhibitory  circuit. 

Consider  an  iV-neuron  winner-take-all  circuit,  with  input  currents  h  >  h  > 
. . .  »  Jjv-  As  shown  in  Equation  1  in  the  main  text,  the  output  voltage  V\  is 

Vi  =  Kln(y-)+V0lnA,  (Cl) 

A 0  J-0 

while  V%  . . .  Vjv  are  approximately  zero.  The  output  does  not  represent  the  input 
ordering  I2  »  I3  »  ...  »  In]  the  largest  input  wins,  and  all  other  inputs  lose. 
We  can  operate  the  circuit  in  another  regime,  however,  which  allows  inputs  Jj . . .  Ij. 
to  win,  and  inputs  Ik+i  ...In  to  lose,  where  the  magnitude  of  Ik  is  under  exter¬ 
nal  control.  Voltage  outputs  V\ . . .  Vk-\  are  now  binary  representations,  while  V* 
maintains  a  logarithmic  encoding  of  the  input  current  J*. 

In  previous  analysis  in  this  paper,  we  used  ideal  current  sources  to  represent 
Ii ...  In •  In  Figure  Cl,  we  replace  these  ideal  sources  with  transistor  realizations. 


Figure  Cl.  Winner-take-all  circuit,  with  transistors  realizations  replacing  ideal  input  current 
sources. 

Transistors  Tz±...TsN,  when  operating  in  the  subthreshold  region,  realize  ideal 
current  sources  if  V^—Vk  >  2V0.  Recall  our  input  Ii  »  1%  »  . . .  »  In,  and  consider 
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the  effect  of  increasing  the  value  of  current  source  Ie.  As  shown  in  Equation  Cl,  the 
neuron  output  Vi  increases  with  Ic.  For  large  Ic,  transistor  T^t  is  no  longer  operating 
in  the  subthreshold  region.  In  this  case,  the  equation  ICl  =  k'(W/L)(Vi  —  Vc  —  Vy)2 
describes  T21,  where  W  and  L  are  the  width  and  length  of  Tfcj.,  and  k '  and  Vy  (the 
threshold  voltage)  are  fabrication  constants.  We  can  solve  for  V\  for  this  situation, 
as 

v'  =  v°lnfy+{*Wir)+vT-  <C2> 

If  we  increase  Ic  further,  Vi  continues  to  increase.  For  a  sufficiently  large  Ic,  V\  can 
approach  V id-  In  this  situation,  begins  to  turn  off,  and  no  longer  acts  as  an 
ideal  current  source  supplying  I\.  In  this  case,  we  can  model  T^1  as  an  independent 
current  source,  supplying  the  current  Ia  =  k'(W/L)(Vdd  -  Vc)2,  as  shown  in  Figure 
C2. 


Figure  C2.  Winner-take-all  circuit,  after  modeling  a  saturated  neuron  with  the  independent  current 
source  Ia. 

To  a  first  approximation,  Figure  C2  shows  a  (N  —  1)  neuron  winner-take-all  circuit, 
with  an  effective  control  current  of  Ic  —  I a. 

We  can  apply  this  technique  to  represent  multiple  input  intensity  scales.  Recall 
the  input  condition  I\  »  1%  »  . . .  >•  In,  and  the  desired  behavior  of  outputs: 
V\ . . .  Vjfc_!  to  be  binary  on,  V*  to  maintain  a  logarithmic  encoding  of  the  input 
current  Ik,  and  all  other  output  voltages  to  be  approximately  zero.  To  produce 
this  behavior,  we  simply  increase  Ia,  until  Vi . . .  Vt_i  are  approximately  Vdd,  but 

Vk  <  vdd. 
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